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Context: The Foundation’s Postsecondary Strategy 
and Every Learner Everywhere 


The Every Learner Everywhere network aims to help institutions of higher 
education address achievement gaps for first-generation students, low- 
income students, and students of color by improving teaching and learning 
with the support of adaptive tools. The network was created in 2017 to bring 
together the capabilities of its 12 member organizations to implement a Bill 

& Melinda Gates Foundation strategy of working with two- and four-year 
colleges to promote effective implementations of adaptive courseware in 
high-enrollment “gateway” courses.* This work is one component of the 
foundation's larger Postsecondary Success strategy “to ensure that all students 
who seek the opportunity are able to complete a high-quality, affordable 
postsecondary education that leads to a sustaining career.” Making progress 
toward the foundation's goal will require U.S. colleges and universities to 
transform themselves in ways that enable them to raise college completion 
rates for an increasingly diverse population of college goers. This means better 
serving first-generation, low-income students and students of color. Every 
Learner Everywhere network activities address a key aspect of postsecondary 
transformation—the improvement of teaching and learning. 


Why gateway courses? Difficulties that many first-generation students, low-income 
students, and students of color encounter in the introductory courses they take in their first 
two years of postsecondary education can lead to discouragement and abandonment of 
college plans (Bloemer, Day, & Swan, 2017). This insight led the foundation's postsecondary 
team to focus on gateway courses. If students have higher success rates in these courses, 
they will accumulate more credits toward graduation in their freshman year and be more 
likely to persist in their academic program (Adelman, 2006; Doyle, 2011). 


Why adaptive learning? Learning systems and instructors provide students with adaptive 
learning experiences when they tailor instruction to the individual needs of different 
students. Research has demonstrated that learning is enhanced when instruction adapts 
to (1) students’ prior knowledge levels, (2) their learning strategies and errors, (3) student 
affect and motivation, and (4) differences in students’ ability to regulate their own learning 
(Aleven, McLaughlin, Glenn, & Koedinger, 2017). Digital learning systems can not only 
provide differentiated experiences to different learners but also give learning data to 
instructors. Instructors can use that data to identify students who need additional support 
and to pinpoints topics with which many students are struggling so that they can return to 
those topics in class. 


1A gateway course is the first credit-bearing college-level course in a discipline or program of study. 
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The foundation's Postsecondary Success Strategy calls out adaptive 
courseware as a key lever for improving students’ course outcomes and, 
through those impacts, student retention and attainment. As a result, the 


key performance indicator for the Every Learner Everywhere network's 

work with an initial set of college and university partners was stipulated 

as “demonstrating that when there is implementation support for use 

of adaptive courseware in gateway courses, the percentage of students 
(including the percentage of low-income and minority students) completing 
the gateway courses for a program of study rises by 15 percent (including for 
low-income minority students) by 2022.” 


Why lighthouse institutions? 


The first set of colleges and universities served by the Every Learner Everywhere network 

is intended to produce learning about what it takes to implement adaptive courseware 
well, and to spread best practices around both courseware use and the use of objective 
evidence to improve instruction broadly within a higher education institution. The learning 
that will be gained from intensive work with the initial set of postsecondary institutions will 
offer both examples of improvement in diverse settings and a concrete set of practices and 
tools that other higher education institutions can use with less involvement and support 
from the Every Learner Everywhere network organizations. 


What does it mean to scale an education innovation? 


One of the key elements of the Gates Foundation’s Postsecondary Success strategy is 

the high-quality implementation of adaptive technology in gateway courses at scale. But 
defining and measuring scale in this context is complex: Unlike discrete products that are 
bought and sold, educational innovations have multiple components, and each component 
may be present to a greater or lesser degree (rather than simply present or absent) 

and may be strong in some parts of the institution but not in others. Head counts and 
product purchases are inadequate to capture the complexity of spread for an educational 
innovation. Moreover, there is neither a single actor (like a buyer) nor a discrete event 

(such as point of sale) that fully captures the experience of the intervention. Northwestern 
University Professor Cynthia Coburn (2003) articulates the case for a multidimensional view 
of scale. Her conceptualization includes dimensions of the innovation’s depth (change 

in underlying pedagogical beliefs and practices), sustainability, and a shift in ownership 
(such that the education institution takes responsibility for and makes decisions about the 
innovation which is no longer viewed as belonging to an external organization) as well as 
spread (which includes looking at penetration within each educational unit, not just the 
number of participating units). 
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Why a network? 


The Every Learner Everywhere network is one of several solution networks the foundation 
has funded as part of its larger strategy to create an ecosystem of supports for higher 
education transformation and improvement.* The foundation chose to fund networks 

of service providers, rather than any single provider organization, because networks 
promote breadth of capacity, flexibility, collaboration, and redundancy. Coalescing multiple 
organizations with a variety of strengths and capacities around a common goal enables 

a network to address challenges that no single individual or organization could handle 

by itself. It also builds collaborations among groups that do not normally interact and 
promotes the flexibility needed to respond rapidly to changes in the field. As the various 
solution networks expand and become interrelated, the foundation expects an ecosystem 
supporting the transformation of higher education institutions will emerge. 


The Lighthouse Institutions 


During the first phase of the Every Learner Everywhere work, network members Achieving 
the Dream (ATD) and the Association of Public and Land-grant Universities (APLU) worked 
with two-year colleges and four-year universities, respectively, to develop proof points 
for the value of the supports the network could provide for effective implementation 

of adaptive courseware in gateway courses. The plan for this work entailed inviting, 
convening, and advising faculty and staff teams led by a senior administrator from a 
dozen lighthouse colleges interested in using adaptive courseware. These lighthouse 
colleges were recruited from three states (Florida, Ohio, and Texas). APLU and ATD 
solicited applications to become an Every Learner Everywhere lighthouse institution from 
institutions they had worked with in the past. To be a lighthouse institution, a college 
needed to commit to: 


e« Convening a cross-functional project team, including a department chair or dean, a 
senior representative from academic affairs, faculty in selected departments, and staff 
responsible for instructional technology and faculty professional development, to 
redesign one or more gateway courses to incorporate adaptive courseware; 


e Sending 3 or 4 participants to a series of Every Learner Everywhere events 
including a teaching and learning summit; 


¢ Participating in an online community of practice around teaching and 
adaptive courseware; and 


e Engaging in project research conducted by Digital Promise, including faculty surveys, 
interviews with faculty project team members, and assembling anonymized student 
outcome data for students taking the courses undergoing redesign. 


In return, the college would receive a modest subgrant ($25,000 or $50,000) and technical 
support services from APLU or ATD, as well as from Digital Promise. Seven community 
colleges signed on to work with ATD, and five four-year universities committed to APLU 


? Other solution networks deal with student advising and developmental education. See https://Wwww.naspa. 
org/press/naspa-announces-the-advising-success-network and https://strongstart.org/. 
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for Every Learner Everywhere activities during 2019. Key characteristics of these lighthouse 


institutions, including their prior experience with adaptive courseware, are shown in 


Table 1. Readers should be aware that the institutional data in Table 1 are taken from the 


Integrated Postsecondary Education Data System (IPEDS) records for undergraduate 


enrollments. IPEDS does not include data on the number of first-generation college goers, 


who comprise a large proportion of the students at the lighthouse institutions. In addition, 


the lighthouse institutions serve large numbers of part-time students from low-income 


backgrounds, who are not eligible for Pell grants because of their part-time status, and 


hence the percentage of Pell recipients as shown in the table underrepresents the low- 


income enrollment at these institutions. 


Table 1. Characteristics of Every Learner Everywhere lighthouse institutions 


Institution 


Type Enrollment? % Students 
of Color? 
Amarillo 2-year 9,854 47% 
College 
Broward 2-year 40,784 63% 68% 
College 


Cleveland State | 4-year 11,999 
University 


| 
Cuyahoga 2-year 23,440 32% 
Community 

College 


Houston 2-year 57,200 63% 53% 
Community 
College 


Indian River 2-year 16,686 41% 60% 
State College* 


% 


Prior Experience with Adaptive 
Courseware 


Some staff familiar with 
adaptive learning products, but 
lack deep understanding. 


Broward had used ALEKS for 
the first 6 weeks of a course to 
get students “up to speed.” 


Adaptive courseware had 
been used in undergraduate 
mathematics and chemistry 
instruction, and the College 
of Science and Health 
Professions had indicated 
interest in broadening use 
beyond these departments. 
The Math Emporium, which 
uses Courseware, raised 
pass rates in developmental 
courses from 48% to 70%. 


One faculty member had 
extensive experience using 
different adaptive courseware 
products, and several others 
had some experience but 
needed more information 
about products available and 
their quality. 


A few individual instructors 

had used adaptive courseware 
as one-offs, but no systematic 
or large-scale implementation. 


Not familiar with adaptive 
course products beyond 
ALEKS (where they 

weren't using the adaptive 
component). No systematic 
process for selecting 
courseware or digital learning 
tools. 


Digital Promise 
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Institution Type Enrollment? | % Students of | % Pell Grants | Prior Experience with 
Color? Adaptive Courseware 


Lorain County 2-year 10,644 20% 52% No systematic process for 

Community selecting digital learning 

College* tools. Any course revisions 
and use of adaptive 
courseware were isolated and 
individual faculty-driven. Had 
used other adaptive software 
in the past, but with ending 
of that software contract, did 
not work to update use with 
new software. 


Miami Dade 2-year 54,973 84% 74% Use of adaptive software or 

College* other technology to some 
degree was common but 
varied greatly based on the 
faculty member. 


University of Familiar with customized 

Central Florida* adaptive rather than off-the- 
shelf adaptive courseware 
products. 


University of 4-year 21,464 85% Only a handful of instructors 
Texas at El Paso used adaptive learning 
(Business). No systematic 
process for selecting 
courseware and other digital 
learning tools. 
University 4-year 24,678 90% Adaptive courseware had 
of Texas Rio been used in mathematics 
Grande Valley* and chemistry. Use of 
courseware mainly to create 
more consistency across 
sections. 


University of 4-year 16,065 16% Adaptive courseware (ALEKS) 

Toledo used in first-year mathematics 
sequence as part of a 
state-supported initiative 
to use co-requisites rather 
than remediation in math. 
Adaptive courseware used for 
placement in chemistry, and 
to some extent in an optional 
co-requisite course, but 
not in the gateway courses 
themselves. Faculty using 
adaptive courseware were not 
fully utilizing its capabilities. 


* Member of the foundation's Frontier Set 

2 Total undergraduate enrollment in fall 2018 from the Integrated Postsecondary Education Data System (IPEDS). 

© Percentage of African American, Hispanic, Native American and Pacific Islander students in fall 2018 enrollment from IPEDS. 

¢ Percentage of full-time, first-time, degree/certificate-seeking undergraduate students receiving Pell grants in 2017-18 from IPEDS. 
Note: An additional four-year institution, Florida International University, will begin working with APLU in 2020. 
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Needs Assessment 


Following early conversations with each lighthouse institution and an initial site visit, 

staff from their technical assistance provider (ATD or APLU) and Digital Promise met to 
discuss where the institution was in terms of key capacities for supporting effective use of 
adaptive learning at the start of their engagement with Every Learner Everywhere. These 
needs assessments were structured as ratings on seven dimensions that the Every Learner 
Everywhere team judged important for implementations of adaptive learning: 


e Alignment: To what extent is the use of adaptive courseware to improve gateway 
course success rates for low-income and underrepresented minority students viewed 
as essential to meeting the institution's strategic goals? 


Leader Support: How strong is campus leader support for improving teaching and 
learning, implementing adaptive learning, and closing gaps in success rates? 


Project Team Capacity: To what extent is there a project team on board with the 
needed capacity (i.e., expertise and available time) for redesigning the selected 
gateway course(s) for cycle 1, supporting effective implementation of a blended 
course, and designing and implementing an impact study? 


Capacity for Course Redesign Using Adaptive Learning at Scale: Across the 
institution, how much experience and instructional design expertise is available to 
work on course redesign incorporating adaptive learning? Specifically: 


° To what extent does the institution implement practices for regularly reviewing 
and refining its gateway courses? 


° To what extent are there staff members familiar with adaptive learning products 
in multiple subjects? 


° Is there a systematic process for selecting courseware and other digital 
learning tools? 


Capacity for Using Data to Support Continuous Improvement of 
Teaching & Learning: 


° Across the institution, how common is it to analyze student learning and 
course success rates? 


° Do department chairs know the student success rates for their gateway courses? 
° Do departments have access to data from the institutional research office? 
° Is evaluation expertise readily available? 


° Is there support for using data from digital learning systems formatively to 


improve instruction? 
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e Capacity to Support Faculty Development: 


° Across the institution, to what extent are there professional learning opportunities 
(such as workshops, mentorships, communities of practice, and training sessions) 
available for both adjuncts and regular faculty and instructors? 


° Are incentives (e.g., mini-grants, release time) available to support faculty and 
staff involvement in redesigning courses to incorporate adaptive courseware and 
promote equitable learning outcomes? 


° Do these supports attend to issues around quality implementation of adaptive 
learning, including leveraging data available from learning systems? 


e Capacity for Inclusive Teaching Practices 


° Across the institution, to what extent are departments and faculty encouraged and 
supported in efforts to improve teaching and learning for low-income and under- 
represented minority students? 


° Do department chairs and faculty Know the success rates in their gateway courses 
for key student subgroups? 


° Are faculty and staff being guided by research on pedagogical approaches that 
reduce equity gaps? 


For each dimension, the institution received a rating from 1-4 from their technical 
assistance provider and Digital Promise liaisons, with 4 being the most mature instantiation 
of the rubric. Table 2 provides the range and average values for the seven dimensions 
among the 12 lighthouse institutions. 
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Table 2. Every Learner Everywhere assessment of lighthouse institutions’ baseline capacity for 
redesigning courses with adaptive courseware at scale 


Capacity Dimension Rating Description of Average Institution 
at Project Start 


Alignment “Adaptive learning/courseware" 
and “improving introductory 
courses” rarely appear in 


institutions’ strategic plans. 


Leader Support Good, but project lead at some 
institutions had limited influence 


institution-wide. 


Project Team Expertise Project teams initially included 
most but not all of the needed 


types of expertise. 


Course design activities were 
siloed and optional at most 
campuses. Typically, there was no 
systematic process for examining 
the quality of gateway courses. 


Using Data for Improvement 


Systematic processes for using 
data to improve teaching and 
learning were largely absent. 
Learning system data were rarely 
used for improvement purposes. 


Faculty Development Professional learning 
opportunities for faculty were 

of uneven quality and not 
incentivized; there were none for 


adjuncts. 


Inclusive Teaching Practices Faculty were unaware of course 
success rates for different kinds of 
students; they had not received 


training in inclusive pedagogy 


Designing Courses with Adaptive 
Courseware 


With one notable exception invited into the cohort as a model of advanced practice 
(University of Central Florida), the lighthouse institutions were missing some of the 
capacities needed to use adaptive courseware strategically to improve outcomes for 
low-income students and students of color when the project started. Institutional strategic 
plans, for example, did not include implementation of adaptive courseware. Even though 
the strategic plans talked about improving teaching and learning or closing achievement 
gaps, they did not highlight gateway courses as a focus for improvement efforts or adaptive 
learning as a catalyst for instructional improvement. The project strategy was to deliver 
technical assistance in a way that supported the institutions’ capacity building. 
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For the Every Learner Everywhere work, each lighthouse institution had a project lead 

who assembled a project team with all or most of the kinds of expertise identified in their 
agreement with their technical assistance provider. However, only a few teams talked about 
a campus-wide strategy linking the implementation of adaptive learning to a commitment 
to closing gaps in course success rates for different kinds of students. 


Most of the lighthouse institutions had staff with instructional design expertise who were 
available to help faculty design or redesign courses for the Every Learner Everywhere effort. 
However, before engaging with Every Learner Everywhere, these teaching and learning 
staff had worked with individual faculty members who happened to seek their help, and 
they were not consistently leveraged by academic departments in efforts to 

improve instruction. 


Also absent at many of the institutions were regular, systematic departmental processes for 
examining the quality of gateway courses. Administrators were generally aware of course 
success rates (the proportion of students enrolled in a course who received an A, B, or C), 
but most did not have institution-wide practices around looking at these data on a regular 
schedule and developing action plans in response to them. For their part, faculty knew the 
proportion of students succeeding in their course overall, but did not know success rates 
for specific kinds of students, such as African American and Latinx students. Without this 
kind of information, many faculty did not fully appreciate their role in a larger effort around 
equity and student success. Moreover, some faculty attributed low success rates in their 
course to qualities of their students (e.g., inadequate preparation or poor work habits) and 
did not feel they had the power to increase student learning and success rates through 
modifications to their teaching practices. 


Professional learning opportunities to help faculty acquire new teaching strategies and 
learn to use digital tools were offered on most campuses, but participation was voluntary 
and typically not incentivized. On most campuses, professional development opportunities 
were not available at all for adjunct instructors, who tend to change from term to term but 
are increasingly teaching the majority of gateway course sections. 


Finally, most faculty had not received any professional development around culturally 
responsive instruction and pedagogies that are particularly effective with low-income 
students and students of color. 


This composite portrait represents the baseline of lighthouse institutions with respect to 
capacities needed to leverage adaptive courseware for student success. The Every Learner 
Everywhere supports for lighthouse institutions were designed to help them build these 
capacities and practices. The technical assistance team will examine where the lighthouse 
institutions stand with respect to the same dimensions at the conclusion of the spring 2020 
academic term. 
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Description of Every Learner Everywhere 
Technical Assistance 


To make headway in improving gateway course success rates for underrepresented 
students, Every Learner Everywhere needs to reach large numbers of institutions to help 
them improve their capacities to enhance the quality of gateway courses. The rationale for 
starting with a relatively small number of lighthouse institutions is to: 


¢ Gain new insights into best practices in implementing gateway courses using active 
and adaptive learning; 


¢ Catalyze the emergence of a cohort of higher education leaders and faculty with 
experience supporting the implementation of course redesign and continuous 


improvement processes; 
¢ Better understand what external supports are both valuable and scalable; 


¢ Codify practices and develop tools for supporting course redesign and 
improvement efforts at scale; and 


¢ Generate compelling examples of course redesign efforts that improved 
student outcomes. 


The support activities supplied by Every Learner Everywhere were designed to help the 
initial cohort of 12 institutions become “lighthouses” that could guide the way for additional 
institutions to make the journey more rapidly and efficiently. These support activities 
recognized the expertise and leadership already available in the lighthouse institutions, 

and drew upon the experience of a set of eight universities that had worked with APLU on 
implementing adaptive courseware under a prior grant. 


The main components of the technical assistance were: 


¢ Face-to-face work at the institution, including one- or two-day workshops, 
bringing in consulting faculty from institutions experienced in implementing 
adaptive courseware. Lighthouse institutions had one to three Every Learner 
Everywhere on-campus visits during 2019. 


e Coaching at a distance through scheduled conference calls (five to eight, 
depending on schedules) with their technical assistance provider and calls 
with other higher education institutions implementing adaptive learning in 
the same academic disciplines. 


¢ Convenings and campus visits that brought the lighthouse institutions 
together with others either experienced in implementing adaptive learning 
or at a similar stage of launching their initial efforts (two to four of these events 
depending on institutional preferences). 
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Interaction with Digital Promise was woven into these activities. A Digital Promise 
researcher accompanied the technical assistance provider on one of the early on-campus 
visits to each lighthouse institution to promote the use of a data-informed continuous 
improvement process in conjunction with the institution's efforts to improve teaching 

and learning. The researcher also explained the Every Learner Everywhere process for 
collecting student-level course success data before and after course redesign and helped 
project teams plan the designs for their course redesign impact studies. The latter effort 
involved balancing the methodological elements of a rigorous research design with the 
realities of campus culture and the limited time and capacity for conducting research in 
parallel with redesign and implementation efforts. Digital Promise researchers described 
design alternatives and the benefits of more rigorous designs, but left the design decision 
to the lighthouse institution team. In most cases, the course lead chose to compare 
student grades and success rates (proportion finishing the course with a grade of A, B, or C) 
for fall 2019 sections experiencing the adaptive version of the course to the same measures 
for sections that had the nonadaptive version with the same instructors in the prior year. 


Digital Promise also worked with institutional research offices to do any necessary tailoring 
of Digital Promise’s data request to the institution's student information system and to 
explain the process for de-identifying student data and doing secure data transfer. Finally, 
Digital Promise researchers observed and documented interactions within and across 
lighthouse institution teams attending Every Learner Everywhere convenings. 


Courses Redesigned for Fall 2019 


Because of a later-than-anticipated project start (several months into 2019 for most 
lighthouse institutions), two of the 12 lighthouse institutions felt they could not complete 
their course redesign work in time to pilot the new adaptive version of their courses in fall 
2019. The 10 institutions that did pilot adaptive courses in 2019 launched 41 redesigned 
courses in aggregate. These courses involved seven subject areas (Biology, Chemistry, 
Economics, Language Arts, Mathematics, Physics, and Psychology) and a range of 
products, including ALEKS, Inquizitive, Lumen Waymaker, Pearson Mastering Biology/ 
Physics, Pearson MyMathLab, Smart Sparrow eSpark, and Wiley Plus. Table 3 shows the key 
characteristics of the lighthouse institution courses involved in fall 2019 pilots. 
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Table 3. Fall 2019 Adaptive Course Pilots 


Institution Type Course Subject Area | Number of | Estimated 


Instructors? | Number of 
Students 


2-year Composition | English 
and II 


Amarillo Community 2-year College Math 
College Algebra/ 
College Algebra 
for STEM 


2-year Introductory Chemistry 
Chemistry | 


Amarillo Community 2-year Gen Organic Chemistry 
College & Biological 
Chemistry 


2-year Principles of Chemistry 
Chemistry | 

2-year Principles of Chemistry 
Chemistry II 

2-year Adult Education | English 
& Literacy 


4-year Applied Algebra | Math 


4-year Precalculus | Math 
4-year Precalculus II Math 


4-year The Living Biology 
World 

4-year Introductory Biology 
Bio | 

4-year Introductory Biology 
Bio Il 

4-year General Chemistry 
Chemistry | 

4-year General Chemistry 
Chemistry II 

4-year Organic Chemistry 
Chemistry | 

4-year Organic Chemistry 
Chemistry II 


Cuyahoga Community 2-year Introduction Biology 
College to Biological 
Chemistry 


2-year Anatomy & Biology 
Physiology 


Cuyahoga Community 2-year Introduction Chemistry 
College to Inorganic 
Chemistry 


Cc 
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Amarillo Community 
College 
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Institution Type Course Subject Area Number Estimated 
Instructors? | Number 
Students 
Cuyahoga Community 2-year General Chemistry 2 57 
College Chemistry | 


Cuyahoga Community Principles of Economics 1 40 
College Microeconomics 
Cuyahoga Community 2-year Beginning Math 4 124 
College Algebra 
Cuyahoga Community 2-year General Physics Physics 1 
College ll 
Cuyahoga Community 2-year General Psychology 2 264 
College Psychology 
Houston Community 2-year Macroeconomics | Economics 2 214 
College 
Houston Community 2-year College Algebra Math 4 203 
College 
Houston Community 2-year Math for Math 3 178 
College Business 
Indian River State College | 2-year Intermediate Math 4 1,427 
Algebra 
Indian River State College | 2-year Quantitative Math 511 
Reasoning 
Indian River State College | 2-year Intro to Chemistry 
Chemistry 
Lorain County Community | 2-year Introduction to Math 
College Statistics 


Miami Dade College 


University of Toledo 4-year Chemistry for Chemistry 
Health Sciences 

University of Toledo 4-year Elementary Chemistry 
Chemistry 

University of Toledo 4-year General Chemistry 4 
Chemistry 


University of Central 4-year College Physics! | Physics 1 
Florida 

University of Central 4-year Spanish 1 Languages 2 
Florida 

University of Texas - Rio 4-year College Algebra | Math 

Grande Valley + Co-Requisite 

University of Texas - Rio Elementary Stats | Math 5 
Grande Valley + Co-Requisite 


Note: UK = Unknown: institution did not provide enrollments from fall 2018 
? Total is number of instructor course implementations; some instructors taught more than one course piloting adaptive courseware. 


5 
2 226 
5 
10 


1 
3 
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We computed prior-year course success rates for the courses undergoing redesign with 
Every Learner Everywhere using data submitted by the institutional research offices of 
participating lighthouse institutions. Prior course success rates varied considerably across 
the courses identified for redesign, ranging from 39 percent to 90 percent with a weighted 
average of 66 percent. The inclusion of courses at the high end of this range of success 
rates indicates that not all lighthouse institutions were limiting this work to their gateway 
courses with the highest DFWI rates. 


Breaking out the data for students of color (African American, Latinx, Native American, and 
Pacific Islander students), we found an average course success rate of 61 percent for these 
41 courses, as shown in Figure 1. When success rates for these students were compared 
to those of white and Asian students in the same courses, the average difference was 

14 percent. As data from the fall 2019 pilots come in, we will estimate the gap in course 
success rates for these student groups when their courses use adaptive courseware. 


Figure 1. Course success rates for African American, Latinx, Native American/Pacific Islander 
students compared to White and Asian students. 
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After computing success rates for the versions of these courses using adaptive courseware 
in fall 2019, Digital Promise also will compare them to the prior success rate for each 
course to quantify the impact of lighthouse institutions’ early efforts at adaptive instruction. 
Digital Promise will hold collaborative impact data review sessions with the lighthouse 
institution project teams to model the process of using both course data and instructor 
experience to inform refinements of the way adaptive courseware is implemented. 


The same process of data collection and impact analysis will be applied to courses 
redesigned with Every Learner Everywhere support and implemented in spring 2020. 
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Instructor Perceptions of Redesigned Courses 


Lighthouse institutions were asked to provide contact information for instructors teaching 
the courses redesigned with Every Learner Everywhere support and for other instructors 
at their institution who taught the same or similar courses, but were not involved with 

the adaptive learning effort. Digital Promise contacted these individuals and invited them 
to take an online survey; a $35 gift card in recognition of the value of their time was 
offered. Digital Promise received 61 survey responses from instructors at nine of the 10 
institutions fielding adaptive courseware in fall 2019. Of these, 39 respondents were using 
adaptive courseware in conjunction with their institution's involvement with Every Learner 
Everywhere (for a response rate of 45 percent among courseware using instructors on the 
survey roster). 


Most of the instructors responding to the survey used the courseware in a blended 
learning approach. Typically 36 to 65 percent of students’ learning in the course was 
expected to occur through working with the courseware. The great majority (90 percent) 
of these instructors had volunteered to use adaptive courseware, and 83 percent chose the 
product they used. A majority (63 percent) of these instructors had used the product in the 
past, though perhaps not in the same way. 


Figure 2 shows the role instructors reported having courseware play (they could 
indicate more than one role). Skills practice, assessment, and conveying content were 
the dominant roles for courseware in these classes. With respect to the latter role of 
conveying course content, most instructors viewed the courseware as a supplement to 
(rather than replacement for) lectures and textbooks. 


Figure 2. Role of adaptive courseware in fall 2019 pilot courses 


Role(s) the adaptive courseware Most important role the adaptive 
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A majority of instructors reported that they first introduced material in class and then 
had students use portions of the courseware on the same topic (60 percent). Only 23 
percent reported having students read about or practice content using the courseware first 
before working on the same material in class (i.e., “flipping the classroom’). 


On average, instructors using adaptive courseware reported that 29 percent of a 
student's grade in their classes was based on his/her work with the adaptive courseware. 
The courseware could count in a number of different ways: 69 percent of instructors 
counted the number of modules completed toward a student's grade; 66 percent counted 
students’ performance on quizzes within the courseware toward grades; 20 percent 
counted the amount of time the student spent on the courseware; and 17 percent used 
course examinations integrated into the courseware. 


Many researchers believe much of the value of adaptive courseware stems from the 
instructor's use of the learning data available from the system as a guide to identifying 
students who need additional support and to focusing future classroom-based activities 
on topics many students struggle with (Kinshuk & Kumar, 2018). The technical assistance 
provided through Every Learner Everywhere stressed the value of using the learner data 
available on instructor dashboards because past studies have found that many instructors 
fail to use it (and sometimes don't even realize it's available). Two-thirds (66 percent) of 
courseware-using instructors responding to the Digital Promise Instructor Survey said 
they looked at the courseware whole-class dashboard once a week or more. Half (50 
percent) said they looked at dashboards for individual students once a week or more. 


A large majority of the instructors who used adaptive courseware in fall 2019 Every 
Learner Everywhere pilots (89 percent) believed using the product had improved student 
learning. Percentages of instructors who agreed or strongly agreed that the courseware 
contributed in various ways are shown in Figure 3. 
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Figure 3. Instructor perceptions of the contributions adaptive courseware made 
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Nearly all of the surveyed instructors (90 percent) said that if given the choice, they would 
use the product again. The Net Promoter Score for adaptive courseware was +.24 with 
twice as many promoters as detractors. 


The courseware is working very well. Those who are 
using it are doing well. | would like help on figuring 


out how to engage those who are not using it... 


- Instructor write-in response 


Digital Promise also collected qualitative data on instructor perceptions of their experience 
integrating adaptive learning into their courses. These data were collected through end- 
of-semester phone interviews and, in one case, a written response offered by an instructor 
who terminated use of the adaptive courseware mid-way through the semester. 


In end-of-year interviews, instructors reported both positive and challenging experiences 
implementing adaptive courseware. Some of the interviewed instructors had previous 
experience with courseware and used the grant to integrate courseware into more sections 
of a course or into new courses, including newly created co-requisite modules. As part 

of this implementation process, they selected topics in the courseware, determined 

the frequency and quantity of problems to assign on each topic, set up quizzes, and 
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decided how courseware assignments and quizzes should factor into grades. In addition 
to integrating the adaptive courseware, some instructors redesigned other components 
of their course. For example, they considered written assignments, the optimal number 
of class hours, attendance policies, use of supplementary instruction (SI) and tutoring, 
whether these supports should be required or optional, and what criteria should be used 
for recruiting undergraduate tutors. 


Instructors found it valuable to have the opportunity to learn how to use features of the 
systems. A few instructors had concerns about the quality of the courseware they selected; 
one instructor decided to withdraw from the project team at her college because she 
found the accuracy of content and user experience of the product she had 

selected unacceptable. 


“On-demand personalized practice opportunities from a product that also 
provides quick, actionable feedback to both students and instructors sounds 
fantastic! Knowing what technology can make possible, | was excited to give 
[PRODUCT] a try. Unfortunately, the more time | spent working with this 
product last semester, the more surprised at the lack of quality | became, and 
the less comfortable | felt with the thought of continuing to make [PRODUCT] 
a required element of my classroom. ... 


“It seems that students with weaker general skills (e.g., reading, problem 


solving, technology savvy) are penalized more by the flaws of the [adaptive 
courseware] product than students with stronger general skills. ... Adaptive 
technology in general has the potential to help close equity gaps, but my 
impression is that the [adaptive courseware] product in its current form may 
even do the opposite by further discouraging students who are already at risk.” 


- Instructor who dropped the courseware product 


Others felt they were getting more familiar with features in the software, and identified 
ways they could implement it better the next semester. Instructors recognized that adding 
adaptive courseware was only one of the elements of instruction they had changed. 
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“| feel like | havent yet had a chance to take as much advantage of the 
feedback that we get from [PRODUCT] and the feedback that we could get 


from all the data from the homework assignments and the adaptive follow- 
ups that are available. So that’s kind of my goal for the second semester 
now that everything's set up and | don’t have the learning curve in front 

of me, can! take more advantage more quickly of the data that’s available 
from using the various software and Clicker Quizzes.” 


- Instructor interview comments 


Student Perceptions of Redesigned Courses 


Digital Promise offered to run online focus groups for interested lighthouse institutions 
piloting courses in fall 2019, and six opted for this data collection. The focus group 
protocol was reviewed by the relevant institutional review boards (IRBs), and instructors 
sent invitations to their students to contact Digital Promise to volunteer for a focus group. 
The process was designed in such a way that instructors would not know which of their 
students had volunteered, and participating students received a $25 gift certificate from 
Digital Promise. 


The student focus groups were conducted online by a Digital Promise researcher or 

an advanced social sciences graduate student. Recordings of the focus groups were 
transcribed, leaving out any mention of a student's name, and transcripts were coded by 
two Digital Promise researchers using nVivo software. A total of 34 focus group participants 
were drawn from five lighthouse institutions (two four-year and three two-year). Focus 
group facilitators estimated that three-quarters of the focus group participants were 
students of color (primarily Latinx and African American). In aggregate, the focus group 
students had experienced seven different Every Learner Everywhere redesigned courses 
involving seven different adaptive products in five subject areas.* Subject areas for the 
courses discussed in focus groups were biology, economics, mathematics, 

psychology, and statistics. 


Students in focus groups reported using the courseware for an average of approximately 
four hours per week, with individual students reporting usage times ranging from 
one to 10 hours. 


5 We were not able to get IRB approval at one four-year institution in time to conduct focus groups, and 
one two-year college conducted its own student focus groups. Data from the latter are not included in the 
tabulations below. 
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“Oh yeah, ona scale of one to 10, | want to say I'm an eight now. Before | 
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experience, but we didn’t really have much sciences. So, trying to do Bio in 
my undergrad, it was just horrible. | didn’t want to ruin my GPA, so | had to 
stop, but now with [PRODUCT] and having all these resources like SI and all 
that stuff, it really is even more confidence.” 


- Student at 4-year institution 


On the whole, focus group students were very positive about the courseware, as shown in 
Figure 4. 


Figure 4. Focus group student perceptions of the adaptive courseware 
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“| think [the courseware] helps me learn [the course subject matter] better 
than | normally would have because it answered a lot of the questions that 


someone would have if they had a passion for it. Some of my classmates, 
you could tell some of them really wanted to be there, and they were excited 
about it and I'd be able to sit down and have a conversation with them... so | 
think it really helped us, not just me individually, but as a class to be able to 
sit there and read up and explore on the different topics of psychology that 
[PRODUCT] provides.” 


- Student at 2-year institution 


Reservations about the courseware were less frequent than positive comments but did 
highlight some issues that deserve attention. Almost half of the students had examples of 
situations where the courseware and the other content in the course were not aligned. 
These inconsistencies bother students, and are a particular impediment for students who 
lack confidence in the subject area or who come to the course with some weaknesses in 
their preparation. 


Some students also expressed the desire for having more interaction with their course 
instructor, something that varied depending on how instructors implemented their 
courseware product and in-class sessions. About a third of focus group participants 
reported experiencing some problem with software glitches or crashes, something 
students need to be prepared for even if the course will be using commercial off-the-shelf 
products. Finally, about one in eight students reported having had some difficulty finding a 
device or internet connection for using the software. Despite progress closing the Digital 
Divide, any teaching and learning effort using digital learning tools should have strategies 
for making sure all students in the course can access the courseware on demand. These 
findings are summarized in Figure 5. 
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Figure 5. Focus group student concerns about the adaptive courseware implementation 
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“My professor didn't go over any problems or anything. So, | think it 
would've been cool if she did at the beginning. Because, | kind of went 
in expecting the problems to be very similar to what we had had for our 
homework and they were very, very different and a lot more challenging, 
and their wording and everything was a little bit different.” 


- Student at 2-year institution 


“| think | could have done the same, | think even better if | didn’t use 
[PRODUCT] because | think | would prefer to have had a class with maybe 
assignments and stuff, where | could actually write and | could go up to the 
professor if | needed help and stuff. Yeah, | prefer to have a professor there 
and assignments to be handed out instead of just the computer.” 


- Student at 4-year institution 
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Lighthouse Institution Reflections on the Process 


Digital Promise researchers conducted semi-structured interviews with lighthouse 
institution project leaders in December 2019 to capture their perspectives on the work 
happening within their institutions. We asked these team leaders to describe their 
institutions’ motivations for entering into the Every Learner Everywhere collaboration. The 
desire to close achievement gaps and interest in improving teaching and learning with 
digital tools generally were common motivations while interest in adaptive courseware 
was not. The decision to enter this particular project was also influenced positively by 
their prior experience with and trust in their primary technical assistance provider (ATD 

or APLU). Finally, team leads told us that the project's provision of some external funds 
was important, even though the amount of funding ($25,000 or $50,000) was modest. 
Institutions were able to use these funds to offer stipends or course releases to faculty 
participating in the course redesign and piloting efforts. Several institutions obtained 
internal funding to further support the effort, but team leads felt it would have been difficult 
to obtain enough internal support without the external contribution. 


The aspects of the Every Learner Everywhere technical support that team leads found 

most useful involved face-to-face interaction, either during on-campus visits or at the ATD 
Teaching and Learning Summit. One team lead at a four-year university described the latter 
experience as “transformative” for her faculty who attended the meeting in giving them a 
better appreciation of their students’ lives and challenges. 


“When faculty members [at the ATD Teaching and Learning Summit] heard 
community college people talk about the real lives of their students, they 
realized those were also the lives of their own students. It was persuasive to 
them that going to college now is vastly different [than when they went], and 


for other people it’s different from their own experience.” 


- Lead for a 4-year university team 


Team leads appreciated the amount of collaboration between Every Learner Everywhere 
network organizations and their campus team built into the experience and the willingness 
of the Every Learner Everywhere team to be flexible around processes in response to their 
institutions’ local constraints, goals, and capacities. Still, there were two major concerns 
about the process: 
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e First, the schedule calling for implementing redesigned courses in fall 2019 after 
starting the project just the prior spring was said to be too rushed. Team leaders had 
to work with whatever faculty and courses could be ready in time, and the process of 
supporting these faculty could not be as deliberate and comprehensive as the team 
leads would like. 


e Second, the project was scheduled to end in June 2020, and team leaders realized 
they still had much to do in terms of refining the adaptive courses they launched in 
fall 2019 and sharing the course redesign process beyond the initial courses and 
faculty participants. 


That said, lighthouse team leaders felt there had been tremendous progress on their 
campuses and that the work will meet their goals for getting involved by project end. Areas 
of progress they highlighted were: 


e Faculty learning about adaptive courseware and instruction 
e Faculty attitude shift toward more responsibility for student learning 


e Changes in instructional practice, not only in implementing adaptive learning but also 
in using more active learning strategies 


¢ Collaborations across instructors working together on the same course(s) 
e Deeper analysis of what they're teaching 
e More consistency across sections taught by different instructors 


¢ Collaboration across different units of the college that don't usually work together 


“| think adaptive is a way to sneak some of these discussions about pedagogy 
because it is something that someone else has already done for you 
generally speaking. | don't think we're going to have a bunch of people who 
are just going to go design their own system. You can implement something 
that someone else has already done, so that makes it a little easier. But then 


it does start to get you thinking differently about how to deliver content, 
which is where we need to get.” 
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“It’s beneficial to have an opportunity for faculty to really test out a product, 
for them to get used to it, [and determine if] this is a model that could be 
used in other situations where we piloted a technology like this and with a 
few faculty and adjust from there. ... Historically when we've done course 


redesign, it’s really like a person who's done that as part of a sabbatical 
project or whatever else, and that person is the one who does things. It’s in 
isolation, so they pick the textbook and they design the class and they pilot 
the software. And if they like it, that’s what is done. But guess what? That's 
okay, maybe, for a small course where very few people teach it or maybe just 
one individual section of something, not for something where we're offering 
45 sections.” 


- Lead for a 4-year university team 


Team leaders do not have the course redesign impact study data yet, but several expressed 
optimism that it would be positive even though their faculty were in their first attempts at 
implementing active and adaptive learning. 


Leaders were mixed in their judgments of whether the work they were doing with Every 
Learner Everywhere had enough momentum to continue after their grant comes to an end. 
Three team leaders expressed confidence the work would continue and several thought 
that at least one of their departments would continue the work, but others would let the 
work fade once the grant drew to a close. A project lead from a two-year institution that 
involved a large number of courses and disciplines in the effort provided the most 
heartening response: 


“Even if we stopped our involvement as a cohesive group next month, 

| think [the effort to redesign courses to leverage adaptive courseware] 
would continue in all disciplines. A few faculty members might drop off, but 
all the disciplines have progressed and are prepared to keep going. Even 
those not thrilled with the interactive product they've chosen are looking to 


change things [and try it again]. We'll convene the SLC [the faculty learning 
community they started for this project] as long as faculty want to participate. 
It will maintain now with or without grant funding.” 


- Lead for a 2-year college team 
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Taking Stock of the Early Lighthouse Experience 


After nine months of working with a dozen lighthouse institutions, we have formed some 
tentative conclusions that can inform future Every Learner Everywhere work: 


¢ Intensive, collaborative work, as conducted with the lighthouse institutions, did 
catalyze changes in practices within these institutions. The institutional changes with 
respect to improving teaching and learning involved only a minority of the institutions’ 
faculty, but team leaders are thinking about how to spread the practices more widely. 


e The structure of the Every Learner Everywhere engagement required a lot from both 
project leads and faculty. Laying out requirements clearly in contracts between the 
institution and their primary technical assistance provider gave the project leads a 
sense of urgency and catalyzed the involvement of multiple college staff who don't 
usually work together. 


¢ “Technical assistance” is something of a misnomer for the engagements between 
Every Learner Everywhere network organizations and the lighthouse institutions. The 
engagement was a series of collaborative planning and coaching activities occurring 
within the general structure stipulated in the grants to institutions, rather than a 
one-way transmission of knowledge from Every Learner Everywhere organizations to 
lighthouse institutions. It was important to have real-time back-and-forth discussions 
as project leads and faculty were planning and starting to implement their 
redesigned courses. 


e In seeking faculty participants, project leads turned to those already onboard with 
focusing on the quality of instruction. They believe that convincing less-interested 
and reluctant faculty through modeling of good practice by these volunteers and 
finding positive impacts on their students is more likely to succeed than 
top-down mandates. 


¢ Both project leads and faculty really appreciated opportunities to meet with field-alike 
peers at other institutions engaged in using adaptive courseware. 


In summary, the level and nature of engagement between Every Learner Everywhere and 
the lighthouse institutions were strong enough to catalyze implementation of adaptive 
learning and data-informed, equity-oriented improvement processes. Participating 
lighthouse faculty increased their understanding of adaptive learning and how adaptive 
courseware functions. Many of these faculty used the course redesign effort as an 
opportunity to think more deeply about their curriculum and instruction, with an emphasis 
on getting more active learning into face-to-face class sessions. Moreover, there are 
indications that some of the lighthouse teams have internalized the key elements of their 
interactions with Every Learner Everywhere (identifying equity gaps, leveraging adaptive 
courseware, supporting faculty learning, and measuring impacts for students) as something 
they need to do for all their gateway courses. Still, the lighthouse teams also realize they 
are still at the beginning stage of making the courses they're working on as effective for 
different student groups as they should be, and that there is a long road ahead to spread 
these practices more widely throughout their institutions. 
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